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DESCRIPTION 
Data set visualization 

The present invention relates to the field of digital imaging. In particular, the present 
invention relates to a method of visualizing a multi-dimensional data set, to an image 
processing device, to imaging systems and to a computer program for performing a 
visualization of a multi- dimensional data set. 

Both endoscopy and biopsy are important techniques in medical diagnosis. It is often 
only possible to determine the origin of a symptom or a sign of an early phase of disease 
by visually examining the inside of the living human body or by taking tissue probes 
from the inside of the living human body. Every year thousands of endoscope proce- 
dures are carried out. Optical endoscopy is a medical procedure used to examine hol- 
low organs or a cavity inside a human body. This is often a painful operation in which 
the endoscope is inserted into the patient through a natural hole or through a small inci- 
sion. 

During such an examination, a medical procedure may be performed which, for diag- 
nostic treatment purposes, removes a sample of an object of interest, such as a piece of 
tissue of a specific internal organ by inserting a medical device into the patient's body 
and, at the same time, monitoring the movement of the medical device by means of, 
e.g., an ultrasound imaging system. In such real-time three-dimensional imaging, the 
choice of the visualization method and orientation is crucial for the intervention success. 
In contrast to non-real time tasks, where the volume may be displayed over and over 
again in review mode, in real-time tasks there is only one chance to display the volume 
in the most useful manner. 

Volume display is a challenging task due to the large amount of three-dimensional in- 
formation that has to be displayed on a screen in two dimensions. All visualization 
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methods of three-dimensional data ignore much of the acquired information. For exam- 
ple, with surface shading many image voxels are hidden in the image, which is dis- 
played on the screen. Similarly, the maximum intensity projection, on each projected 
ray only the contribution of one voxel is displayed. 

5 

One question occurring is what to ignore and what to show in real-time applications, 
since user control may not be possible or may put an undue burden on the user, since 
the physician is fully concerned with operating his interventional devices. 

10 It is an object of the present invention to provide for improved visualization of multi- 
dimensional data sets. 

According to an exemplary embodiment of the present invention as set forth in claim 1, 
the above object may be solved by a method of visualizing an multi-dimensional data 
15 set, wherein the method comprises the steps of performing a segmentation of a structure 
in the data set and performing a visualization of the data set, wherein a projection direc- 
tion of the visualization is determined on the basis of the structure. 

In other words, a structure in a data set may be identified by a segmentation procedure 
20 and, depending on the segmented structure, an image slice or projection of the data set 
is visualized, wherein the position of the projection inside the data set is related to the 
structure. 

Advantageously, according to this exemplary embodiment of the present invention, this 
25 may allow for an automated choice of parameters of the visualization procedure and 
therefore for an improved visualization. 

According to another exemplary embodiment of the present invention as set forth in 
claim 2, the visualization is performed on the basis of visualization parameters which 
30 comprise information about the projection direction and which are determined on the 
basis of at least one of the segmentation of the structure in the data set and low-level 
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analysis of the data set. Furthermore, the visualization parameters are selected from the 
group consisting of a relative position of the structure, a direction relative to the struc- 
ture, a distance between the structure and an object of interest, a motion estimation, and 
a motion compensation. 

Advantageously, according to this exemplary embodiment of the present invention, im- 
age slices may be visualized in a certain direction relative to the structure and, for ex- 
ample, at a certain distance from the structure. Furthermore, according to this exem- 
plary embodiment of the present invention, the multi-dimensional data set may com- 
prise data relevant for motion estimation or compensation, such as, for example, electro- 
cardiogram (ECG) data. This additional data may be used, e.g., for a motion compen- 
sated visualization of the data set. 

According to another exemplary embodiment of the present invention as set forth in 
claim 3, the structure is one of a biopsy needle and an endoscope probe, wherein a first 
projection of the data set is performed in a direction of a longitudinal axis of the struc- 
ture, resulting in a first image with an image surface area perpendicular to the direction 
of the longitudinal axis. Furthermore, a second projection of the data set is performed 
in a direction perpendicular to the longitudinal axis of the structure, resulting in a sec- 
ond image comprising the structure. 

Advantageously, according to this exemplary embodiment of the present invention, this 
may provide for a first image slice visualizing a viewing direction of a biopsy needle 
and a second image slice visualizing the biopsy needle itself and data on a plane com- 
prising the biopsy needle. This may allow for an effective visualization of information 
required for performing a successful and fast biopsy or endoscopy. 

Another exemplary embodiment of the present invention is set forth in claim 4, wherein 
at least one of the visualization parameters is displayed during visualization of the data 
set. 
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Advantageously, this may provide a physician operating a biopsy needle or an endo- 
scope with important information about, for example, the relative position of the struc- 
ture with respect to an object of interest, 

5 According to another exemplary embodiment of the present invention as set forth in 
claim 5, the method further comprises the step of varying a rendering method in an im- 
age resulting from the visualization of the data set, wherein the variation of the render- 
ing method causes a non-uniform quality of the image. Furthermore, according to an 
exemplary embodiment of the present invention as set forth in claim 6, the variation of 
10 the rendering method comprises a variation of a sampling rate in the image and the 
variation of the rendering method is performed on the basis of the visualization pa- 
rameters. 

Advantageously, according to the exemplary embodiments of the present invention as 
15 set forth in claims 5 and 6, this may allow for maximum image quality in the centre of 
an image slice (where the biopsy needle may be pointed to), and, at the same time, for a 
reduced image quality at the edges of the image slice (which are not of high interest to 
the user). Advantageously, this may provide for a reduction of computational cost. 

20 According to another exemplary embodiment of the present invention as set forth in 

claim 7, the segmentation is performed on the basis of one of a Hough Transform and a 
determination of active localizers. 

This may allow for an effective and reliable segmentation of the structure. 

25 

According to another exemplary embodiment of the present invention as set forth in 
claim 8, the data set is acquired by means of an ultrasound imaging system, a CT im- 
aging system and an MR imaging system. 



30 



According to another exemplary embodiment of the present invention as set forth in 
claim 9, an image processing device for visualizing a multi-dimensional data set is pro- 
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vided, comprising a memory for storing the data set and an image processor adapted for 
performing the following operation: loading the data set; performing a segmentation of 
a structure in the data set; and performing a visualization of the data set. The projection 
direction of the visualization is determined on the basis of the structure. 

Advantageously, this may allow for an improved visualization. 

A further exemplary embodiment of the image processing device as set forth in claim 
10. 

The present invention also relates to imaging systems comprising a memory for storing 
a multi-dimensional data set and an image processor adapted for performing a visuali- 
zation of the data set. According to an aspect of the present invention, the imaging 
system is one of an MR imaging system, a CT imaging system, and an ultrasound im- 
aging system. The imaging systems according to the present invention are set forth in 
claims 11 to 13. 

Advantageously, this may allow for an improved visualization of a multi-dimensional 
data set acquired by a CT imaging system, an MR imaging system or an ultrasound im- 
aging system. 

The present invention also relates to a computer program, which may, for example, be 
executed on a processor, such as an image processor. Such computer programs may be 
part of, for example, a CT scanner system, an MR scanner system or an ultrasound im- 
aging system. The computer programs, according to an exemplary embodiment of the 
present invention, are set forth in claim 14. These computer programs may be prefera- 
bly loaded into working memories of image processors or general purpose computers. 
The image processors are thus equipped to carry out exemplary embodiments of the 
methods of the present invention. The computer programs may be stored on a computer 
readable medium, such as a CD-ROM. The computer programs may also be presented 
over a network, such as the WorldWideWeb and may be downloaded into the working 
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memory of an image processor from such networks. The computer program according 
to this exemplary embodiment of the present invention may be written in any suitable 
programming language, such as C++. 

5 It may be seen as the gist of an exemplary embodiment of the present invention that an 
intervention (caused by a user) to an object of interest is visualized without the require- 
ment of an interactive input by the user, which may be a physician. In fact, parameters 
for a visualization procedure, such as, e.g., of viewing direction, may be automatically 
chosen during data acquisition which may allow for an efficient tracking of the actual 
10 orientation and relative position of a structure, such as a biopsy needle, with respect to 
the object of interest, such as, for example, a cyst inside the abdomen or a plastoma or 
neoplasm inside the uterus of a patient. 

These and other aspects of the present invention will become apparent from and eluci- 
15 dated with reference to the embodiments described hereinafter. 

Exemplary embodiments of the present invention will be described in the following, 
with reference to the following drawings: 

20 Fig. 1 shows a simplified schematic representation of an embodiment of a magneto- 
resonance (MR) imaging system according to the present invention. 

Fig. 2 shows a schematic representation of an interventional biopsy visualized by an 
ultrasound imaging system according to an exemplary embodiment of the present in- 
25 vention. 

Fig. 3 shows a schematic representation of projections visualized according to a method 
of the present invention. 

30 Fig. 4 shows a flow-chart of an exemplary embodiment of a method of visualizing a 
multi-dimensional data set according to the present invention. 
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Fig. 5 shows an exemplary embodiment of the present invention of an image processing 
device according to the present invention, for executing an exemplary embodiment of a 
method in accordance with the present invention. 

Fig. 1 shows an exemplary embodiment of the present invention of an MR imaging 
system according to the present invention. With reference to this exemplary embodi- 
ment and to the embodiment depicted in Fig. 2 (ultrasound imaging system), the present 
invention will be described for the application in the field of medical imaging. How- 
ever, it should be noted that the present invention is not limited to the application in the 
field of medical imaging, but may be used in other applications, such as, for example, 
any other form of minimal invasive procedures which may arise, for example, in the 
field of baggage inspection to detect hazardous material, such as explosives, in items of 
baggage or other industrial applications, such as material testing. 

The MR scanner system comprises coils 210 which are arranged along an axis 218 and 
surround an examination space 217, in which a patient 215 who has to be examined or 
from whom a tissue probe has to be taken is positioned. Advantageously, the patient 
lies on a movable table or conveyor belt 216, which is disposed at the lower part of the 
examination space 217. The system of coils 210 surrounding the examination space 
217 comprises an HF-coil 219, an actively shielded arrangement of gradient coils com- 
prising an inner coil 213 and an actively shielded coil or shield 212 and a cryostat 211, 
in which the coils are arranged in order to be cooled down during generation of the 
magnetic field. The arrangement of gradient coils 213, 212 may be connected to a gra- 
dient amplifier 220. 

Furthermore, the MR scanner or imaging system may comprise a motor control unit 
with respective motors, for example, for moving the conveyor belt 216, in the calcula- 
tion unit (not shown in Fig. 1). 
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The calculation unit may be realized by an image processor integrated into an image 
processing device comprising a memory for storing a data set and may also be adapted 
to perform a visualization of the data set according to an exemplary embodiment of a 
method according to the present invention. The data processor or image processor ac- 
5 cording to an aspect of the present invention may be adapted for loading the data set for 
performing a segmentation of a structure in the data set. Furthermore, the data proces- 
sor may be adapted for performing a visualization of the data set, wherein a projection 
direction of the visualization is determined on the basis of the structure. 

10 Furthermore, the calculation unit may be connected to a loudspeaker (not depicted in 
Fig. 1) to, for example, automatically output of an alarm. 

Fig. 2 shows a schematic representation of an interventional biopsy visualized by an 
ultrasound imaging system according to an exemplary embodiment of the present in- 

15 vention. A biopsy needle 205 is inserted into the patient through a small incision 208 in 
order to take a tissue sample from the abdomen 206. Biopsy needle 205 may be oper- 
ated by operating means 201 for cutting a piece of tissue probe 207 from the abdominal 
wall. In order to track the movement of the biopsy needle 205 and its relative position, 
with respect to the object of interest 207, an ultrasound imaging system 202 may be 

20 employed, according to an exemplary embodiment of the present invention. The ultra- 
sound imaging system 202 may comprise an ultrasound transducer used for generating 
ultrasound signals 204 propagating towards the object of interest 207 and to receive 
echo signals reflected from the object of interest 207. Such ultrasound imaging systems 
are well-known in the art and will not be described here in greater detail. 

25 

It should be understood, that, according to the present invention, the imaging system 
202 does not necessarily have to be an ultrasound imaging system but may be any other 
appropriate kind of imaging system, such as, for example, an MR imaging system or a 
CT imaging system. 

30 
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In any case, the imaging system 202 provides for the acquisition of a multi-dimensional 
data set of the object of interest 207. During the operation the multi- dimensional data 
set may be constantly updated. The multi- dimensional data set may be a three-dimen- 
sional data set representing the volume of the abdomen, or a four-dimensional data set, 
comprising additional information, where, for example, a heart rate by means of an 
electro-cardiogram for motion detection and motion compensation. Furthermore, it may 
also be, for example, a four-dimensional data set formed by a succession of three-di- 
mensional data sets. But it should be understood that the present invention is not lim- 
ited to three-dimensional or four-dimensional data sets, but as well may be implemented 
for visualization of five-dimensional or even higher dimensional data sets comprising 
information beyond the scope of three-dimensional volume information and ECG in- 
formation. 

V 

% f 

Somewhere in the volume acquired by the ultrasound imaging system, the biopsy needle 
is present. Displaying the surfaces of the organs may not help at all, since the needle 
may be hidden in an organ. Standard projections through the volume can be faint im- 
pressions about the direction of the needle relative to the point where the tissue has to 
be extracted. 

The method according to an aspect of the present invention now segments the biopsy 
needle from the volume, advantageously in real-time, using volume processing methods 
like a Hough Transform. This segmentation may give the position and elongation di- 
rection of the biopsy needle. If the volume is more projected in the direction of the nee- 
dle elongation, it is instantly clear that a needle targets the centre of interest or not. At 
the same time, a section through the volume with the needle lying in the display slice 
may give information about the distance from the needle tip to the target region. 

It should be noted that the concept may be extended to a thick slap orientation, choice of 
surface rendering viewpoint and an integration of active localizers. For example, the 
segmentation does not necessarily have to be performed by a Hough Transform, which 
is a computer vision algorithm that can robustly detect a wide variety of features such as 
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lines, circles and anything else that can be readily parameterized or otherwise cast in 
terms of a discrete popularity algorithm. If, for example, the structure is linear in shape, 
but not visible to the imaging system, since, for example, some of its physical properties 
are similar to physical properties of the surrounding tissue (for example a reflection 
5 coefficient), it may not be advantageous to use a Hough Transform for segmentation. 
Instead, active localizers may be integrated in the structure (e.g. biopsy needle) and the 
integrated active localizers may be individually segmented by means of an alternating 
external electromagnetic field and an appropriate detector. The detection of active lo- 
calizers is well-known and will not be described here in greater detail. After detection 
10 of integrated active localizers, the orientation and position of the structure may easily be 
performed. 

Fig. 3 shows a schematic representation of two projections visualized according to an 
exemplary method according to the present invention. The first projection or image 
15 slice 301 depicts the object of interest 207 inside the abdominal cavity. Image slice 301 
is projected in the direction of the biopsy needle 205 (see Fig. 2), therefore giving the 
user an idea what the biopsy needle 205 is aiming at. The aim of the biopsy needle 205 
may be represented by cross 303. 

20 Image slice 302 represents the object of interest 207 (e.g. a cyst, abscess or neoplasm) 
and the biopsy needle 205. Since the image slice 302 is taken in the plane of the biopsy 
needle 205, the distance between the tip of the biopsy needle 205 and the object of in- 
terest 207 may be calculated from the data set and displayed on the image slice (here: 5 
cm). Both image slices 301 and 302 may effectively aid the physician during the bi- 

25 opsy, therefore, allowing for a fast execution of the biopsy and an accurate extraction of 
tissue from the object of interest 207. 

Fig. 4 shows a flow-chart of an exemplary embodiment of a method of visualizing a 
multi-dimensional data set according to the present invention. The method starts at step 
30 S0 5 after which an acquisition of a data set is performed in step SI, for example, by 

means of a polychromatic source of electro-magnetic radiation generating a polychro- 
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matic beam and by means of a radiation detector detecting the polychromatic beam, 
which is the case in, for example, CT imaging. 

After that, in step S2 a segmentation of the biopsy needle is performed on the basis of a 
5 Hough Transform or a detection of integrated active localizers. Then, in step S3, a low- 
level analysis of the data set is performed and visualization parameters are selected. 
The visualization parameters may comprise a relative position of the structure with re- 
spect to the object of interest, a direction relative to the structure, a distance between the 
structure and the object of interest and a motion estimation and compensation on the 
10 basis of, for example, ECG-data. The motion may be compensated by a generation of a 
motion map and performing a motion compensated reconstruction of the image slice on 
the basis of the motion map. Motion estimation and compensation is well-known in the 
state of the art and will not be described here in greater detail. 

15 The visualization of the data set comprises a first projection of the data set in a direction 
of a longitudinal axis of the biopsy needle, resulting in an image slice with an image 
surface area perpendicular to the longitudinal axis and a second projection in a direction 
perpendicular to the longitudinal axis of the biopsy needle, resulting in a second image 
slice comprising the biopsy needle. 

20 

In other words, particular selected image slices from the data set are visualized during 
the examination or operation of the patient. The image slices are selected automatically 
by the ultra sound imaging system without the need of input from the physician. Ad- 
vantageously, the image slices represent projections in the view direction of the biopsy 
25 needle and in a direction perpendicular to the view direction of the biopsy needle. 

Therefore, the physician always "looks" in the direction of his biopsy needle 205 inde- 
pendent from its actual position relative to the ultra sound source 202 (see Fig2. 2 and 
3). 

30 During rendering of the first and second projections of the data set in step S5, the ren- 
dering method in the image slices may be varied, resulting in a non-uniform quality of 
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the image. The variation of the rendering method may comprise a variation of a sam- 
pling rate of the images or projections of the data sets on the basis of the visualization 
parameters. For example, the area adjacent to the cross 303 in Fig. 3 may be rendered 
with a high sampling rate, resulting in a maximum image quality in the centre of the 
5 image slice 301, wherein the outer area of image slice 301 may be rendered with a 
lower sampling rate, resulting in a lower image quality at the edges of the image slice 
301. Therefore, important parts of the image 301 (namely the centre) are displayed in 
high image quality, whereas parts of lower interest (the outer areas) are represented with 
lower image quality, therefore, resulting in a reduction of computational costs and im- 
10 proved rendering speed. 

After that, in step S6, parameters based on a low-level analysis of the data set, such as, 
for example, the distance between the structure and the object of interest, are displayed 
during visualization of the data set. 

15 

The method ends at step S7. 

Fig. 5 depicts an exemplary embodiment of an image processing device according to the 
present invention for executing an exemplary embodiment of a method in accordance 

20 with the present invention. The image processing device depicted in Fig. 5 comprises a 
central processing unit (CPU) or image processor 151 connected to a memory 152 for 
storing an image depicting an object of interest, such as a piece of tissue of a patient's 
body. The image processor 151 may be connected to a plurality of input/output network 
or diagnosis devices, such as an MR device or an ultrasound imaging system. The im- 

25 age processor is furthermore connected to a display device 154, for example, a com- 
puter monitor, for displaying information or an image computed or adapted in the image 
processor 151. An operator may interact with the image processor 151 via a keyboard 
155 and/or other output devices, which are not depicted in Fig. 5. 

30 Furthermore, via the BUS system 153 it is also possible to connect the image processing 
and control processor 151 to, for example, a motion monitor which monitors a motion 
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of the object of interest. In case, for example, a lung of a patient is subject to the biopsy 
or an endoscopy, the motion sensor may be an exhalation sensor. In case the heart is 
imaged, the motion sensor may be an electro-cardiogram (ECG). 

5 
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CLAIMS: 

1. Method of visualizing a multi-dimensional data set, the method comprising the steps 
of: 

performing a segmentation of a structure in the data set; and 
performing a visualization of the data set; 

wherein a projection direction of the visualization is determined on the basis of 
the structure. 

2. The method according to claim 1, 

wherein the visualization is performed on the basis of visualization parameters 
comprising the projection direction; 

wherein the visualization parameters are determined on the basis of at least one 
of the segmentation and a low-level analysis of the data set; and 
wherein the visualization parameters are selected from the group consisting of a 
relative position of the structure, a direction relative to the structure, a distance between 
the structure and an object of interest, a motion estimation, and a motion compensation. 

3. The method according to claim 1, 

wherein the structure is one of a biopsy needle and an endoscope probe; 

wherein a first projection of the data set is performed in a direction of a 
longitudinal axis of the structure, resulting in a first image with an image surface area 
perpendicular to the direction of the longitudinal axis; and 

wherein a second projection of the data set is performed in a direction 
perpendicular to the longitudinal axis of the structure, resulting in a second image com- 
prising the structure. 

4. The method according to claim 3, 

wherein at least one of the visualization parameters is displayed during 
visualization of the data set. 
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5. The method according to claim 1, further comprising the step of: 

varying a rendering method in an image resulting from the visualization of the 

data set; 

wherein the variation of the rendering method causes a non-uniform quality of 
5 the image. 

6. The method according to claim 5, 

wherein the variation of the rendering method comprises a variation of a sam- 
pling rate in the image; and 
10 wherein the variation of the rendering method is performed on the basis of the 

visualization parameters. 

7. The method according to claim 1, 

wherein the segmentation is performed on the basis of one of a Hough Trans- 
15 form and a determination of active localizers. 

8 . The method according to claim 1 , 

wherein the data set is acquired by means of one of an ultrasound imaging sys- 
tem, a CT imaging system, and an MR imaging system. 

20 

9. An image processing device for visualizing a multi-dimensional data set, the image 
processing device comprising: 

a memory for storing the data set; 

an image processor adapted for performing the following operation: 
25 loading the data set; 

performing a segmentation of a structure in the data set; and 
performing a visualization of the data set; 

wherein a projection direction of the visualization is determined on the basis of 
the structure. 

30 
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10. The image processing device according to claim 9, 

wherein the structure is a biopsy needle; 

wherein the visualization is performed on the basis of visualization parameters; 

wherein the visualization parameters are determined on the basis of at least one 
of the segmentation and a low-level analysis of the data set; and 

wherein the visualization parameters are selected from the group consisting of a 
relative position of the structure, a direction relative to the structure, a distance between 
the structure and an object of interest, and a motion estimation. 

wherein a first projection of the data set is performed in a direction of a 
longitudinal axis of the biopsy needle, resulting in a first image with an image surface 
area perpendicular to the direction of the longitudinal axis; and 

wherein a second projection of the data set is performed in a direction 
perpendicular to the longitudinal axis of the biopsy needle, resulting in a second image 
comprising the biopsy needle. 

11. An imaging system, comprising: 

a memory for storing a multi-dimensional data set; 

an image processor adapted for performing a visualization of the data set, 
wherein the image processor is adapted for performing the following operation: 
loading the data set; 

performing a segmentation of a structure in the data set; and 
performing a visualization of the data set; 

wherein a projection direction of the visualization is determined on the basis of 
the structure. 

12. The imaging system according to claim 11, 

wherein the structure is a biopsy needle; 

wherein the visualization is performed on the basis of visualization parameters; 
wherein the visualization parameters are determined on the basis of at least one 
of the segmentation and a low-level analysis of the data set; and 
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wherein the visualization parameters are selected from the group consisting of a 
relative position of the structure, a direction relative to the structure, a distance between 
the structure and an object of interest, and a motion estimation. 

wherein a first projection of the data set is performed in a direction of a 
5 longitudinal axis of the biopsy needle, resulting in a first image with an image surface 
area perpendicular to the direction of the longitudinal axis; and 

wherein a second projection of the data set is performed in a direction 
perpendicular to the longitudinal axis of the biopsy needle, resulting in a second image 
comprising the biopsy needle. 

10 

13. The imaging system according to claim 11, 

wherein the imaging system is one of an MR imaging system, a CT imaging 
system, and an ultrasound imaging system. 

15 14. A computer program for performing a visualization of a multi-dimensional data set, 
wherein the computer program causes an image processor to perform the following op- 
eration when the computer program is executed on the image processor: 
loading the data set; 

performing a segmentation of a structure in the data set; and 
20 performing a visualization of the data set; 

wherein a projection direction of the visualization is determined on the basis of 

the structure. 
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ABSTRACT 

In real-time three-dimensional imaging the choice of the visualization method and ori- 
5 entation is crucial for intervention success. The key question is what to ignore and what 
to show in real-time applications, where user control is not appropriate. The invention 
addresses this problem by visualizing an intervention (caused by a user) to an object of 
interest without the requirement of an interactive input by the user. Advantageously, 
according to an exemplary embodiment of the present invention, parameters for a visu- 
10 alization procedure are automatically chosen during data acquisition which may allow 
for an efficient tracking of the actual orientation and relative position of the structure 
with respect to the object of interest. 

(Fig. 3) 
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